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ABSIEACT 

Activi-cies of the Teacher Corps' "Corpis Member 
Training Institute" (CMTI) were studied to illustrate methods of 
inquiry which can be adapted to the study of teacher training events 
to capture some of the reality of these events* For the CMTI, a Study 
Team was mandated to demonstrate the impact of the Institute 
(identify, document, judge, and communicate features) • The Study Team 
interpreted this mandate to mean demonstrate the salient features in 
an evolving, interactive, teacher training event so that the features 
could be adapted for use in other teacher training situations. By 
identifying procedures that captured the patterns of interactions^ 
the reality of the event being examined was reflected. The team 
rejected standard methodologies as not useful because they fail to 
provide adequate information about the dynamics of the event and 
because too little useful information is uncovered that could be used 
in replicating features of the event. They found that a useful 
collaborative technique for identifying the important characteristics 
of teacher training events is a combination of a multilevel data 
gathering approach, a time-series design, field study procedures 
involving observation, interviews, and photography. (MM) 
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The purpose of this paper is to reflect on the activities of the CMTI 
Study Team fron a methodological perspective. It is our intent to draw 
your attention to three points: 

1) That any methodology used by an investigator must first reflect 
the reality of the event being examined • 

2) That the typical methodologies often used to study teacher training 
events are too limited in that they fail to capture the reality of 
such events. 

3) That there are methods of inquiry which can be adapted to the study 
of teacher training events which if used allow one to examine and 
if clever capture some of the reality of such events* 

The Reality of Dynamic Teacher Training Events 

^Tncn collaborating on the study of any phenomena the methodologist must 
first ask \^at are the questions that need to be answered?*' Once those 
questions are determined, the essential characteristics of the phenomena must 
be identified so that a plan for data collection, analysis and documentation 
can be established. 

For the Teacher Corps* "Corps Member Training Institute'* (Cim), the 
policy mandate to the Study Team was to demon*" rate the impact of the 
Institute. The key term, demon strate , implies c* at the Study Team was to 
idenr.ify» to document, to judge, and to communicaL some features of the 
event that were successful or unsuccessful 

The language and meaning of this mandate leaves no doubt that the need 
for demonstration is more than most standard meanings given to program evalua- 
tion in at least two important aspects. First , the responsibilities for 
denonstration include more than the documentation that certain objectives 
were atcained and others were not. Such documentation does not communicate 
enough about the training procedures to other staffs who may want to try 
these procedures in their own settings. Second, demonstration is not merely 
the collection of all program residue* The responsibility for communicating 
specific training procedures and their successes and failures under certain 
program conditions to individuals not involved with the program requires 
more than a collection of all the instructional materials used in the program. 
Yet so often we see investigators when asked to show a training program to 
an outsider respond by collecting their materials. They rely upon an effective 
display to show that the program should be reproduced at another setting. 
That kind of information is both too much and too little. It is too much of 
the content for an outsider to place into a meaningful context and it is too 
little of the rationale, intentions and specific instructional procedures used 
to make this content meaningful. 
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The purpose of demonstration is to coxninunicate to those not originally 
involved with a particular program the salient features of the program. The 
overriding question is: ^^Iiat does an outsider have to know to try some of 
the specific features of the training strategy in another setting? Thus, 
the special challenge of demonstration can be stated as "describing the 
successes and failures of specific features of a training program for possible 
replication at other sites/' The underlying meaning of demonstration is to 
be found in the implications of the phrase ''for possible replication at other 
sites. ' In fact, it is the meaning of th^ word ''replication" that must be 
understood* 

In teacher training v;e never mean replication in a strictly scientific 
sense. No training event can be replicated, if by '^replicated'' we mean that 
t.ie event can be made to reoccur precisely as it once occurred, the 1975 
CMTI, for example, could not be replicated even if one chose the same 
instructors, the same interns, and returned for another month at Richmond, 
Nor does the word "replication" mean adoption. Adoption is to make an outside 
program one's ovm by selection or assent. It implies little or no change of 
the outside program. Rather the word "replication" implies adaptation. 
Adaptation suggests significant change of the original. Adaptation of an 
instructional process or a program is not only to alter some salient features 
of that program but to alter those f-^atures of that program to meet one's own 
needs. The complexity of most educational environments necessarily implies 
that the meaning of program replication is the identification of features 
which can be adapted for use in Other sites. For example, one of the initial 
features of CMTI identified by the Study Team was the "esprit de corps" that 
was created and maintained during the training session. This apparently 
important feature of CMTI could undotibtedly be replicated (adapted) in other 
training settings. 

In summary, when adaptation is expected from urogram demonstration, the 
responsibility of the demonstrator goes beyond^/? do cuiu nting effectiveness and 
displaying program materials. The ^responslbtaflty extends to identifying, to 
describing and to judging the valu^ of features which others may adapt for 
their purposes. The identif ication^of salient features embedded in instruc- 
tional program which may (or n|ay not) be adaptable is not a simple task. In ' 
short, the difficulties of prdgram replication place creative responsibilities 
upon those undertaking program deironstration. 

Characteristics of Dynamic Events . The salient features to be demon- 
strated for CMTI are those which characterize dynamic events. The adjective 
"dynamic" has been deliberately chosen to describe teacher training events 
because they have the following characteristics: 

1) They involve humans interacting with each other. 

2) They occur over time. ^ 

3) They are concerned with some substantive content. 

4) The content is brought out over time through an interactive process. 

5) Organizing 1, 2 and 3 into an event is deliberate In order to have 
effects or outcomes on the participants. 
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Furtherraore, the interactive process is not static. The variety of partici- 
pant responses often changes the content and direction of the event, some 
outcomes are often embedded within the process itself, each event is occurring 
within a larger and complex learning milieu, and the apparent content of the 
event shifts and changes when viewed from different personal and time perspec- 
tives. Presumably, to understand what happens in a training event, one must 
have a description of the event itself which illuminates these characteristics. 
In particular, any event must be considered as a mixture and interaction of 
physical, social and moral-psychological environments. The physical space 
(time, place and objects) in which an event occurs is suggestive, facilitating 
and constraining, with respect to what occurs. What people do and say to 
and with others — both '^direct" observable behavior (naturally occurring "free" 
actions and responses) — are part of every event. And finally, the intentions 
of all the participants shape and give meaning for each individual to each 
event. Salient features of demonstrated events are not definable in a conven- 
tional sense. Features of dynamic events are planned and have a definite 
structvire but the embedded patterns of interactions cannot be entirely 
anticipated. 

The methodologist^s task is to identify procedures which capture the 
patterns of interactions ia order to describe the course of events and predict 
effects. He must use theoretical constructs of social interactions, or opera- 
tional categories based on an empirical examination of the event to identify 
patterns. And apain, effects are not just terminal outcomes; effects include 
how ideas develop and change over time. 

In summary, our first p6int was that the methodology used must reflect 
the reality of the event being examined. For the CMTI Study Team this was 
interpreted to mean that we were to demonstrate the salient features of an 
evolving, interactive, teacher training event o that they could be adapted 
by others for use in their teacher training site 'tions# 

The Inadequacy of Standard Evaluati on Methods 

Standard nethodolog^es are not useful for demonstrating teacher training 
events because they are too limited. They fail to provide adequate information 
for policy decisions, they provide insufficient data about the dynamics of 
the event for an understanding of the event, and they provide too little useful 
information which one could use in replicating features of an event. 

Although the information presented here is not new, we felt obligated 
to point out these inadequacies for tV7o reasons: first, in the past, 
evaluators of Teacher Corps projects have primarily used such methods; and 
second, it has been our experience that many teacher educators are unaware 
of the limitations of standard evaluation methodologies. 

Fox (1976) in analyzing and summnrizing prior evaluations of Teacher 
Corps projects has characterized those as employing either a "one shot case 
study" preexperimental design or a **one group pretest, posttest" preexperi- 
raental design (Campbell and Stanley, 1963). These are two of a set of 
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conventional designs borrowed from an agricultural experlcental paradigm. 
In general, this methodology is based on three key assumptions: (1) treat- 
ment effects are additive, (2) treatment effects are constant, and (3) there 
is not interference between different experimental units. Before explaining 
these assumptions and discussing their consequences, the three terms— treat- 
ments, effects, and experimental units—must be understood* 

Treatment. The objective of most agricultural experiments is to compare 
the yields of a number of plant varieties, fertilizers, or soil characteristics. 
The term "treatment," for example, might refer to the fertilizers being compared. 
For training »vents "treatment" translates to a specific instructional plan* 
Rt:>cently this uas been accomplished by developing instructional modules based 
on specified behavioral objectives with explicit instructional routines. In 
extreme cases these instructional routines are controlled by programmed texts 
or scripts to be followed. 

In reality most training events cannot be so explicitly planned. The 
difference between what is planned and what actually occurs is considerable. 
This dii:ference is both natural and beneficial. Training events are not 
mechanistic routines to be blindly followed. Real events grow, change, and 
develop as the human beings involved in the event interact. In fact, it is 
the actual patterns of interactions, rather than the intended treatments, 
that are the important features of training events for policy makers, other 
researchers and replicators. 

Effects . The term "effects" refers to yield or end product of a treat- 
ment. In education this translates to assessment of performance at the 
termination of the "treatment." This is now most often accomplished by^ 
developing criterion-referenced tests based on the behavioral objectives of 
the treatment r Such terminal assessment captu-^es only intended performance 
outcomes not underlying outcomes or unintended - -tcomes and completely fails 
to capture what happens before, during or after t*. ^ event. Furthermore, too 
often intended performance outcomes are identified cly in terms of new 
concepts to be learned. Such an emphasis on predetermined symbols and their 
manipulation is inadequate for all but the most trivial of human interactions 
with new information. It misses the personal meanings of the new information 
and the integration of these meaning with past (and future) experiences. 
These "other" outcomes are of essential importance for replication. 

Experimental Unit . The term "experimental unit" refers to the soil 
plots or "the smallest division of the experimental material such that any 
two units may receive different treatments in the actual experiment" (Cox, 
1958, p. 2), In educational training events the "experimental unit" thus 
should be the population taking part in the event. However, even though 
most training events are carried out with groups, individual students are 
often used as the experimental units. The consequences of using this 
incorrect experimental unit are disastrous with respect to the three assump- 
tions underlying conventional methodology. 

Treatment Effects Are Additive . The mathematics of determining experi- 
mental effects is based on Equation 1 (Cox, 1958, p. 14). 

1) y = u + t 
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Equation 1 says that the total quantitative effect of yield (y) after a 
treatment can be broken dovm into two subquantities: u — a quantity depending 
only on the particular experimental unit, and t — a quantity depending only 
on the treatment used. 

There are three immediate consequences of this assiimption* The first 
is the quantif lability of all three terms—y, u and t* While it is true that 
one mark of a mature science is the possession of sophisticated measurement 
instruments and techniques, we must admit that at present in education we «re 
not able to quantify with any validity or accuracy many terms in an educational 
setting. ^ For example, one anticipated treatment effect of CMTI was that the 
participants would share in a meaningful multi-cultural experience. The 
photographic essay, ''Together,'* (Tabachnick and Lemes, 1975) was prepared as 
a means of documenting (but not quantifying) the actuality of that experience. 

The second consequence of the additivity assumption allows the possibility 
for adding or subtracting these treatment effects in an algeuraic manner in 
order to remove the quantity depending upon the experimental unit* If a 
''control** group is used that does not receive the treatment, then Equation 2 
shows the total "effect" measured when no treatment is administered. 

2) y = u 
c 

Subtraction of Equation 2 from Equation 1 gives an estimate of the effect due 
only to treatment (see Equation 3), 

y^ = u + t 

- y « u 
c 



3) y, -y, = t 

Equation 3 says that one need only have the measurements of the total quantir 

tative effects after treatment (y. ) and for control group (y ) to estimate 

t c 
the effect due only to treatment (t). 

The third consequence of this additive assumption is the possibility of 
estimating differences between two treatment effects* If measurements are 
taken after two different treatments, they can be subtracted. The following 
set of equations shows this algebraic process* 

y^ = u 4- t^ (Measurement after Treatment 1) 

y2 u + t2 (Measurement after Treatment 2) 
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Equation ^ says that the difference between the final measurements after 
Treatments 1 and 2 (y^ ' considered to also be the difference 

between the two effects due only to Treatments 1 and 2 (t^^ - t2) . 

Note, however, that Equation 3 and Equation A are correct only if the 
effects depending on the unit (the u's) are eqqal for different units* These 
conditions are assumed to be trtie when there is nc systematic bias which 
differentiates the experimental units. Control of bias is usually accomplished 
by random assignments to an appropriate experimental design. In fact, true 
control groups are not often feasible in teacher training situations. E>/en 
where alternate treatment groups are possible (Equation 4), they are only 
helpful in estimation of the program^ s effect if the treatment effect of the 
other treatment were knovm — presumably from another experiment which employed 
a true control group. 

Many investigators use the pretest in the '^pretest-posttest" design as 
a measure of u. 

- y = u 
P P 

5) - y = t 
t p 

The resulting equation (5) indicates that the treatment effect (t) is equal to 
the difference in pretest and posttest scores (gain). Obviously, Equation 5 
is valid only if the unit term is the same before and after intervention. 
However, as Campbell and Stanley (1963, p. 8) -o forcefully point out, there 
are several sources of potential invalidity to Ms assumption such as history, 
maturation, instrumentation, test treatment intet. -:tion, etc* Investigators 
simply cannot blindly use gain scores as estimates treatment effects. 

Note that since control groups have been in use, the designs used to 
evaluate Teacher Corps projects neither actually can be used to estimate 
program effects. 

Also, because of this additivity assumption an evaluator is primarily 
concerned about the measureiaent of specific attributes only at the ^nd of 
treatment. The focus of this concern Is on validity, The inferences of the 
results depend upon the extent to which the measurements are valid. The 
researcher must defend (or is criticized upon) the various ways in which 
validity can be established (construct, content and criterion-related validity)* 
Once validity can be established, results are verified. 

Unfortunately, while validity of terminal assessment Is important, it 
does not address questions about the identification of the treatment as it 
involves the pattern^ of interactions between treatment and subject or the 
extent to whicli the effects of one treatment are actually "added upon" the 
effects of anotn^ treatment. In education, not only are these later concerns 
most crucial to treatment replication but they should also be more open to 
critical inspectioiL^y^ 
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Constancy of Treatment Effec ts* In an attempt to increase the generali«- 
ability of the findings most evaluators replicate the training procedure either 
by using the basic training program in two or more settings, or by assuming 
each individual subject in a single training setting (every subject is a 
replicate of every other subject). In either case, one must assume that treat- 
ment effects are constant. Tliis assumption says that treatment effect does 
not change when the treatment is given to two different units. Algebraically, 
this assumption allows Equation 6 to represent the measured effect after a 
specific treatment on one unit (u, ), and Equation 7 to represent the measured 
effect of the same treatment upon a different unit. 

6) y^ Uj^ + t 

Since treatment effect is assumed to be the same for these two different 
units, the subtraction of Equation 7 from Equation 6 states a very important 
consequence, of this assumption (see Equation 8). 

8) yi-y2 = "i-"2 

Equation 8 states that if tx^o different experiernental units (u^^ and U2) receive 
the same treatment, then the differences in effect (y^ and y2) ^re only due to 

differences in units. With this asusmption, when one compares the end of treat-- 
ment measures, one is also comparing the differences between the experimental 
units. Note, following this argunent, if u^ and were identical units, the 

difference in treatment effects should be zero. However, if aspects of train- 
ing procedures are adapted (not adopted) for t with different populations, 
one would actually anticipate different treatmen effects. That being the 
case, it would not be possible to arrive at Equatx n 7. Thus, the conclusion 
that one is comparing differences between units when one compares measure© at 
the end of the same treatment is false. 

I 

This problom is especially acute v;hen one uses individual students as 
the experimental unit. As students are exposed to new material we expect 
them to assimilate those new ideas into their own personal meanings or id^- 
tional scaffolding. We e;;pect the same training event to have different 
effects on different students. Some will assimilate and use lots of new ^ 
information in one way, others iray generate quite different kinds of new 
information and relationships. Psychologists now are generally agreed that 
unless the influence of individual difference variables is considered, predicted 
outcomes of training events will be marked by within-treatment variation^ 
Persons indeed do differ in how tho'y respond to the same information or the 
same instructional procedures. Thus, the assumption that treatment effects 
are constant is simply false in many teacher training circumstances. 

Lack of Interfere n ce of Experimental Units . This assumption says that 
when more than one experimental unit is used, there is no interference or 
interaction between the units. This assumption is particularly important 
if statistical analysis is to be made of the observations since the statistical 
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are based on an assumption of independenca of observations (or unit measure- 
ments). If the experimental units really were independent classes, then one 
might argue lack of interference between the classes. On the other hand, in 
teacher training events, if students are the units, this assumption is clearly 
false. Such interference between units is the essential interaction between 
human beings one expects in dynamic events. In fact, investigators have 
typically assumed that the treatment effect for a class is simply an aggregate 
of individual effects. The argument is uausally that "we teach pupils, not 
classes." This simply is not the reality of dynamic events. By assuming no 
interaction the researchers closes his eyes to the essence of the event. 

In summary, the basic assumptions underlying the conventional methods 
of inquiry are not met by the reality of dynamic training events. One can 
only conclude that conventional methods of inquiry are not appropriate for 
the demonstration of dynamic training events. 



Process Evaluation: An Alt ernate MethodoloR V 

Identifying the salient features of dynamic events for demonstration will 
depend upon the extent to which an investigator can capture and communicate 
those features. Fortunately, there are methods of inquiry which can be 
adapted to study such events. For example, the problem of evaluating a program 
within a complex learning milieu, has been addressed by Parlett and Hamilton 
(1972) who suggest the concept of "llluralnative evaluation'': an intensive 
examination of the evaluated program through a variety of strategies in order 
to help illuminate significant program features. Another example is the 
work being undertaken by Glass (1975) who is adapting the techniques of time- 
series analysis to the problems inherent in the examination of treatment 
effects over time. A third example Is t)ie "p* -^cess development" model for 
curriclum development and evaluation suggested .v Romberg and Hernandez-Nieto 
(1976) that responds to special evaluation problb s inherent in the nonlinear 
characteristics of instructional processes as well *..s human development. 

What we have done is to combine features from a variety of sources into 
a dynamic, evolving method of inquriy we are calling "Process Evaluation." 
^'Evaluation" because we are indeed attempting "to judge or determine the worth 
or quality of ( Webster's New World Dictionary . 1970, p. 48A). The referent 
of evaluation is a "product . " For CMTI the process of Instruction used in 
teacher training events > "worth or quality" has been defined in terms of 
demonstrable features of training events. The term "process" has been used 
to emphasize the evolutionary processes involved in training events. Such 
events have two main features which haye helped us in developing this alter- 
nate methodology — they are time embedded, and they evolve (have a becomlngness). 
Time is important both as a basis for describing training events (events start, 
progress and end), and as a variable that characterizes events. Becomlngness 
is important since outcomes are not just yield. One must look for intended 
and unintended outcomes. Thus, multivarlate-multilevel information is critical. 
In addition one uiust try to capture how these many outcomes grow and evolve 
over time (including before, during and after the event). Finally, since such 
events Involve humans we must know how they interact — spend time. From these 
considerations the following notions of design, intervention and observations 
were used. 



ERIC 



10 



-9- 



ERIC 



Desicn 

The design used in the CMTI Impact Study is *'time-series/* Campbell 
arid Stanley (1963) drew educators' attention to this design and labeled it 
"the interrupted time-series quasi-experiment They also discussed the 
statistical analysis difficulties associated with that particular design. 
Since 1963 the utility of the "time-series" design has been endorsed in 
several textbooks and articles but rarely actually applied to educational 
research. The reasons for lack of use fall into two categories. First, a 
misunderstanding, somewhat perpetuated by Campbell and Stanley, that the 
intervention was necessarily of short duration (such as the effect of passing 
a law). It can and is most often used in economics to monitor what goes on 
before, during and after a prolonged intervention. And second, the more 
important reason for lack of use has been the difficulties with statistical 
analyses. However, at this time considerable work has been done on time- 
series analyses. The best summary is by Glass, et.al. (1975) in their book 
De sign and Analysi s of Time-Series Experiments . In that work they describe 
eight different time-scries designs* These eight designs are summarized in 
Table 1. 



Insert Table 1 Here 



The varieties of intervention effects which could be examined are listed in 
Table 2. 



Insert Table 2 Hexi* 



The intention of a time-series design is to monitor ongoing events. The 
result of this monitoring is to plot a series of points such as the series of 
points indicated in Figure 1. 



Insert Figure 1 Here 



This series of points (joined for illustration) shows rankings on the five 
perspectives on teacher tasks during CHTI. Sec Figure 2 for a second example 
from CMTI. 



Insert Figure 2 Here 
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Then one examines the graphs to determine either change in level or change 
in direction by using graphical or spectural analysis techniques* These 
are not new' techniques* They have been commonly used In engineering and 
economics for several years. However, to develop data from human beings 
raises several problems. First, the usual required number of points is too 
large (spectural analysis is usually done with no less than 200 data points). 
The second problem is statistical dependence. Human beings being retested 
have a tendency to remember how they answered the same questions rather than 
responding independently. And third, very often in education we are inter- 
ested in multivariate effects rather than univariate effects* To date ho 
multivariate models have been developed for testing effects of time-^serles 
studies. Nevertheless, in CMTI we used a combination of a "single group 
multiple I design" (for the summer training program) and a "stratified multiple 
group single I design'* for their project experience in ;:he field (see FJgur;e 3). 

Insert Figure 3 Here 



Interventions 

These were three interventions: first, the two-week training session on 
organizational characteristics; second, the two-week training session on 
instructional techniques; and third, the on-site training in Teacher Corps 
Projects. These three interventions were not under the control of the CMTI 
Study Team* In addition, while there was an overall instructional plan, the 
training events evolved over time in ways which were not completely anticipated 
by the instructors. 



Observations 

The observations used in this design were of two types. First, objective 
assessment via the use of surveys ideally for an Impact study one would 
develop a multivariate-multilevel objective assessment procedure*. 

Multivariate, in that several variabj.es would be studied simultaneously 
and of necessity treated as independent (because of the limitations of the 
statistical analysis procedures associated* with time-series designs). Since 
it is clear that the set of variables would not be Independent the reporting 
of results must be carefully judged in terms of the overall set of variables. 
The basic battery that was developed and administered by the QITI Study Team 
was multivariate but only at one level. The CMTI survey battery Included 
five instruments: teacher tasks, intern tasks, school success, school fallurej» 
and expectations* On each instrument five dependent variables were Judged; 
namely, teaching as personal development, teaching as technique, teaching as 
curi culum development, teaching as community involvement, and teaching as 
orga lizational behavior. 
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In each of the four suirveys the interns were asked first, to rank order 
each set of five items and second, to determine the value of each of the items . 
to themselves. For example. Figure 4 shows one set oi~~^ive items from the 
tasks of an intern survey. Each item in this set is an exemplar of one of 
the five perspectives. The tasks of an intern survey contains five sets of 
items similar to the set given below. Each survey was then administered 
several times. 



Insert Figure 4 Here 



Multilevel would mean that the data should be gathered at more than one levels 
An initial level would include assessment related to the objectives of instruc- 
tion (usually the concepts and skills one would expect to master during instruc 
tion). At subsequent levels underlying effects would be probed. 

The CMTI Study Team did not collect objective assessment data. The 
instructors of both initial intervention did collect that information. In 
future designs, it would be advantageous if systematic data were gathered 
at this intended outcome level. The instruments developed by the CMTI staff 
were intended to measure a second level of socialization to teaching. 

To reduce the statistical dependence of observations find to reduce the 
amount of testing time Involved in this study, matrix sampling was followed 
to gather the data. A small number of students at each observation time 
were administered a portion of the total battery. Estimates of population 
char:acteriscics were derived from those subpopulations. 

The second part of our data collection scheme was to gather subjective 
assessment data in order to give meaning to the objective data. The objective 
data alone simply yields numbers (points on a graph) but fails to give a clear 
understanding 'of the salient features of the program. Thus, observations, 
interviews^ and photographs were used to give meaning to the dat^. The 
procedures used were adapted from the field study techniques of anthropology- 
sociology. Communicating the results from this study involves knowing the 
interventions, how they evolved, and summarizing the objective and subjective 
data. From this rich data base it is anticipated that consistent patterns 
can be reasonably identified and communicatedi to others. 



Summ ary 

There are methods of inquiry which can be adapted to the study of teacher 
training events which if used would capture some of the reality of such 
events. Hoyever, the methods are not easy to use. We are not claiming we 
have found any panacea to the problems of assessing and evaluating teacher 
training programs. What we are claiming, however, is the need to expand 
one's conceptualization of evaluation for such events to the identification 
of salient features which can be demonstrated. We do believe that the 
combination of a multivariate, multilevel data gathering approach, a time- 
series design and field study procedures involving observation, interview, 
and photography, etc., are useful as a collaborative technique for identifying 
the important characteristics of teaching training events. 
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